Multi-target Knowledge Distillation via Student Self-reflection

Abstract Knowledge distillation is a simple yet effective technique for deep model compression, which aims to transfer the knowledge learned by large teacher small student model. To mimic how teaches student, existing methods mainly adapt an unidirectional transfer, where extracted from different intermedicate layers of used guide However, it turns out that students can learn more effectively through multi-stage learning with self-reflection in real-world education scenario, nevertheless ignored current methods. Inspired this, we devise new framework entitled multi-target via or MTKD-SSR, not only enhance teacher’s ability unfolding be distilled, but also improve student’s capacity digesting knowledge. Specifically, proposed consists three target mechanisms: stage-wise channel (SCD), response (SRD), and cross-stage review (CRD), SCD SRD feature-based (i.e., features) response-based logits) at stages, respectively; CRD encourages conduct self-reflective after each stage self-distillation Experimental results on five popular visual recognition datasets, CIFAR-100, Market-1501, CUB200-2011, ImageNet, Pascal VOC, demonstrate significantly outperforms recent state-of-the-art